Codecov Report

Attention: Patch coverage is 84.09091% with 7 lines in your changes missing coverage. Please review.

Project coverage is 72.37%. Comparing base (c05931a) to head (a84692a).

@@            Coverage Diff             @@
##             main    #2957      +/-   ##
==========================================
+ Coverage   72.35%   72.37%   +0.02%     
==========================================
  Files         131      131              
  Lines       40602    40644      +42     
  Branches    40602    40644      +42     
==========================================
+ Hits        29376    29416      +40     
- Misses       9336     9339       +3     
+ Partials     1890     1889       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@rtyler to my understanding it's not required, but it's assumed when a table has NOT NULL columns

See discussion here: #2882 (comment)

I added this because of the error and this statement in the protocol

If the table is on Writer Version 7, the feature invariants must exist in the table protocol's writerFeatures.

@ion-elgreco I should also mention that this came from a table created by delta-rs and does not contain any actual invariants. I believe newer DBRs and Delta/Spark releases are more strictly adhering to the protocol and expect invariants to be defined on writerVersion 7

The phrasing of the protocol is lousy here. append only, check constraints, and a few other features have similar "to support this feature it must be in the writerFeatures verbiage. But as best as I can tell Delta/Spark is only being pedantic about invariants 😕

(venv) plantain% pwd
/home/tyler/source/github/delta-io/delta-rs/unoptimized-table/table/_delta_log
(venv) plantain% rg "invariant"
(venv) plantain%

I added this because of the error and this statement in the protocol

If the table is on Writer Version 7, the feature invariants must exist in the table protocol's writerFeatures.

It's a bit poorly written, but the sentence starts with "To support this feature". I also misread it before and assumed it was required but it's just to support it in v7 we need to add the feature.

Did you check when you create a v3,7 table with nullable columns in spark what the protocol log states?

Okay so I did some testing with PySpark and I am going to pull this into draft again to modify the changes a bit. Essentially if a non-nullable column exists and things are upgraded to writer7, the invariants writer feature is added, but no invariants are set 😕

This behavior is not specified anywhere in the protocol which is fun 💀

DeltaTable.create(spark).tableName('nonulliz').addColumn('ts', 'TIMESTAMP_NTZ', nullable=False).location('./nonulliz').execute()

{
  "commitInfo": {
    "timestamp": 1729706895482,
    "operation": "CREATE TABLE",
    "operationParameters": {
      "partitionBy": "[]",
      "clusterBy": "[]",
      "description": null,
      "isManaged": "false",
      "properties": "{}"
    },
    "isolationLevel": "Serializable",
    "isBlindAppend": true,
    "operationMetrics": {},
    "engineInfo": "Apache-Spark/3.5.3 Delta-Lake/3.2.0",
    "txnId": "1e0e57c1-d1a2-41d9-9402-53516b8f4c77"
  }
}
{
  "metaData": {
    "id": "721d696f-8396-4a69-851c-23702b8b61bb",
    "format": {
      "provider": "parquet",
      "options": {}
    },
    "schemaString": "{\"type\":\"struct\",\"fields\":[{\"name\":\"ts\",\"type\":\"timestamp_ntz\",\"nullable\":false,\"metadata\":{}}]}",
    "partitionColumns": [],
    "configuration": {},
    "createdTime": 1729706895473
  }
}
{
  "protocol": {
    "minReaderVersion": 3,
    "minWriterVersion": 7,
    "readerFeatures": [
      "timestampNtz"
    ],
    "writerFeatures": [
      "timestampNtz",
      "invariants"
    ]
  }
}